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PREFACE 

This paper vas written during the first part of the Research for 
Better Schools ^3 (RBS) school Improvement effort In the area of citizen 
education vhlch Is being carried out In junior high and middle schools. 
The effort Involves a process of goal setting and related measurement of 
student characteristics, In preparation for program assessment and develop- 
ment. In this effort, RBS has encouraged the school people to construct 
their own locally-valued goals and to determine the substance of the 
related measures. RBS^ role has been to help the school people carry out 
these tasks so that the outcomes will provide a firm basis for the eusulng 
work on the educational program. In keeping with this role orientation, 
the present paper does not promote the choice of particular goals or mea- 
sures; Instead It provides general questions and considerations concerning 
measures In the area of citizen education. 

Although the paper was developed as part of the RBS work, It was ' 
written with more general use In mrndf. It could be used for obtaining 
measures at most grade levels and, with adults, though some suggestions do 
require that the citizen education target group be literate. The paper 
was designed as a guide for any person who takes on the responsibility of 
obtaining measures for citizen education goals, as described In the "Purpose 
and Intended Audience'^ section of the Introduction to the paper. 

The present version of the paper Is a revision of a paper er.itltled 
Conceptualization and Specification of Measures of Student Behavior Related 
t o Citizen Education Goals, written by the present author with the 

111 
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assistance of John J* Bowers* Insights for the revisions came from 
helpful reviews by John Bowers » Donald Coan» Florence Davis » James 
DlCostanzOt Carl 6uerriero» Russell Hill» Richard Hulsart» Louis 
Maguire» James Oswald^ Barbara Presseisen» and Sanford Temkin. Also 
influential in the revision was the author^s firsthand experience in 
the use of related procedures in one school and his coordination of 
use in three other schools* 
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OBTAINING MEASURES OF CITIZEN EDUCATION GOALS 
latroductloa 

The introduction aakes explicit the assumptions underlying the 
g^ildance presented in this paper. In particular^ the intended use of 
the paper» the required expertise of users » the envisioned use of citizen 
education goal measures* and the availability of appropriate measures 
are discussed* in order to alert the reader to the author*s orientation. 
Purpose and Intended Audience 

The purpose of this paper is to present a procedure for obtaining 
■easures of citizen education goals in a systematic and defensible 
manner. ^ Using the procedure does not require a high degree of measure- 
ment expertise; however* experience in and sympathy for the careful use 
of measures in educational settings are needed. Its use does require con*- 
siderable interchange with the person or persons who are responsible for 
identifying and educating for the particular citizen education goal in 

2 

question; hereafter such persons will be referred to as goal experts. 
This paper Is primarily a guide to obtaining the necessary clarifications 
of the goal from the goal expert (s). The outcome of use of the guide 
is a set of explicit Implications for measure specifications* not the 

^Although no definition of citizen education is presented in this 
paper* the goals referred to in the Appendix do indicate the nature and 
breadth of the domain. 

2 

Goal experts in citizen education should ba chosen for their reputa- 
tion for knowledge in the specific goal area; there probably will be 
different goal experts for the different goals. 
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ineasure Itself* 7h« measure Itself will remain to be developed or 
chosen, and would be obtained In accordance with both the statement of 
Implications for measure specifications and specific content supplied by 
the goal expert(s). An overview of use of this paper Is In Table 1, 
Perspective on the Reed for Measures: The Indicator Approach 

The primary need for measures of goals Is to allow for objective 
determination of how well the goals are being achieved. However, there 
are at least two ways to interpret this need, as is revealed in .he 
following discussion. 

As with most major education goals, the goals of citizen education 
are phrased in general terms. Measures, on the other hand, refer to 
specific behaviors* Thus, in obtaining measures of citizen education 
goals, there is a problem of relating the specific behavior evidenced 
the measure to the general characteristics referred to in the goal. 

One solution to the problem is to have content experts analyze the 
general goal into specific objectives, with the final aim of generating 
a set of behavioral objectives that are essentially synonymous with 
particular test items. All the measures so generated are then the 
measures associated with the original general goal. The conclusion 
concerning achievement of the goal is based upon a summary of results 
^^from the many associated measures. This solution is here called the 
behavioral objectives approach * 

A contrast to the behavioral objectives approach is the indicators 
approach . This approach Involves only enough clarification to Identify 
the single or few behaviors most readily associated with the goal. 

2 
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TABLE X 



Outline of Tasks Involved In Obtaining Measures of 
Citizen Education Goals and Related Guidance In This Paper 



Tasks of Obtaining Measures 

1. Adopting a perspective on 
the tasks of obtaining 
measures 



2. For each goal» determining 
the goal*s general category 

3. For each goal» developing 
goal^clarlf ylng questions 
within the goal*s general 
category of 

a. kn0tfledge» or 

b. skill* or 

c. attltud^» or 

d. behavior. 



A. For each goal» obtaining 
goal clarification from the 
relevant goal expert(s) 



5. For each goal» developing 
liE^licatlons for measure 
specifications 



6. For each goal» guiding the 
relevant goal expert(s) In 
developing/selecting measures 

7. Considering the practicality 
and acceptability of the use 
of the measures 



Guidance in the Paper 

"Perspectives on the Need for Mea** 

sures ..." (pp. 2*^5) 
"Finding . . . Indicators in the 

Literature" (pp- 5-6) 

"General Goal Categories" ' (pp . 6-6) 



**Ktiowledge" and related examples 

(pp. 6-10 and 25-31) 
**Skill" and related examples 

(pp. 10*12 and 32*35) 
^*Attltude*^ and related examples 

(pp. 12*14 and 36-AA) ^ 
"Behavior" and related examples 

(pp. 15*17 and 45-49) 

Examples of such clarification in 
the "General clarification" and 
**Relation to citizenship" sections 
of related examples (pp. 25-49) 

As in task 3 above» with special 
attention to the "Implications for 
measure specifications" sections 
of related examples 

Outcomes of the above steps » plus 
references given within categories 
(pp. 10» 12» 14» and 17) 

"Practical Concerns" (pp- 17-16) 
"Prudential Concerns** (p. 19) 
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Examples of Indicators from other areas of social concern are the 
Gross National Product, used as an Indicator of the health of the na- 
tional economy, and body temperature, used as an Ind^.cator of the health 
of the Individual. Using the Indicator approach, the conclusion about 
goal achievemept Is made on the basis of one — or at the most, a few — 
measures. * 

A comparison between the two approaches reveals that the Indicator 
approach Involves less detailed analysis of the goal and provides less 
lofonaatlon to be coasldered In deciding about goal achievement* Thus, 
the Indicator approach may be seen as more efficient. Its major short^ 
coming in comparison with the behavioral objectives approach is that, in 
focusing on only one or a few characteristics, it provides relatively 
little diagnostic information* Thus, the Indicator approach Is rela** 
tlvely poor for planning remediation, which is the next task after conr 
cx'iding that a goal was not sufficiently achieved. 

The particular configuration of relative strengths and weaknesses 
Implies that there are complementary roles for the two approaches* The 
Indicator approach might be used first, as & screening procedure to 
identify priorities among citizen education goals or to determine the 
importance of doing further work related to a particular goal* The 
isore thorough, behavioral objective approach would then be used only for 
the high priority goals, and would provide more detailed results for 
diagnostic prescriptive purposes* 

In this context, the present paper should be seen as relating to 
the first task, because it takes an indicator approach. In order to 

6 
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Supplement the paper with guidance on the behavioral objectives approach* the 
reader should refer to one or more of the general works on the topic (e^g.» 
Bloom» Hasting* & Madaus« 1971; Gagae & Brlggs» 1974; and Mager» 1962) and 
to one specifically on citizen education (Tjart & D'Amlco» 1979)^ 
- Finding Citizen Education Goal Indicators In the Literature 

A probable Initial response to the task of obtaining goal Indica- 
tors Is to try to find an already developed measure that has a title 
or content which matches the lanf.uage used In the goal statement* This 
response Is to be expected^ because many believe (1) that finding an 
already available measure saves the time and effort of developing one» 
and (2) that a measure presented In the literature Is probably better 
than one developed locally* 

These beliefs may be called Into question for several reasons. 
Plrst» the time and effort required to find a measure that matches a 
citizen education goal may well exceed expectations* Aside from related 
knowledge of Information taught In the social studies part of the currlc- 
ulum» iDost areas usually associated with citizen education do not have 
measures listed in standard references. Further » those that were 
found in one search were — with a few exceptions — only borderline 
In terms of technical measurement quality (Sanders^ 1978)* Finally^ one 
aspect of a gocxi measure Is Its relevance to those who will be using Its 
results and there seems to be a resistance among educators to the use 
of measures developed elsewhere (e*g*» Qulnto & McKenna» 1977y^ ' 
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Although these conditions do not Imply that a technically adequate 
and locally acceptable measure will not be found through a search of the 
literature, they do Imply that a development effort will often be needed. 
This paper Is primarily Intended to be a guide for that development 
effort, though It may also be valuable In judging the appropriateness 
of already developed ineasures or choosing Items from such measures. 

The Guide 

Beginning with an Individual citizen education goal, one :an derive 
specifications for the appropriate measure. As elaborated In the follow*- 
Ing pages, the first step Is to determine the general category to which 
the goal belongs: knowledge, skill, attitude, or behavior. This cate- 
gorization Indicates what general measurement procedures will probably 
be most appropriate. Then, within categories, a further clarification 
of the content Implied by the goal Is necessary, so that measurement 
procedures can be more thoroughly specified. 

Guidelines for needed goal clarification are presented In the sec- 
tions of this paper that deal with each of the four general categories. 
Also, examples of such clarifications and related Implications fur 
measure specifications are presented In the Appendix^ 
General Goal Categories 

An Initial step In obtaining an appropriate measure for a goal Is 
to determine the general type of goal. A useful set of measurement proce^ 
(lures Is determined by the answers to two questions about the goal: 



(X) Does the goal refer primarily to capabilities, as In test-like clrcum" 
stances, .^r to naturally occurrlrg characteristics, ^s In **candld camera** 
circumstances? (2) Does the goal refer primarily to what people say or to 
what people do? 

The four combinations of answers to these two questions are presented 
In Table 2 as four general goal categories. In parentheses are labels 
for each category, chosen to refer to the type of psychological character- 
istics usually associated with the category. 



Capability 



Natural 
Characteristics 



TABLE 2 
General Goal Categories 



ffligt People Say 

Presenting Information 
when asked (Knowledge) 



Presenting personal 
perceptions or 
evaluations (Attitude) 



What People Do 

Performing a 
task when asked 
(Skill) 

Normal activity 
(Behavior) 



The primary reason for the use of the above categories Is that they 
lead easily to further distinctive measure procedures. Knowledge Is 
usually measured with questions and answers, with answers scored for 
correctness. SkilJs are assessed with performance tests, which either 
require that a sequence of steps be conducted In a particular order or have 
a definable product as an outcome. As considered here, attitudes ar^ 
ordinarily assessed with scales and Interviews In which persons report 
their perceptions or evaluations. Finally, behavior, as dlsciissed In this 
paper. Is associated ^/Ith observational procedures used by persons other 
than the person whose behavior Is to be noted. 



The four general goal and related measurement procedures are each 
discussed In the following sections of this paper. The sections are 
designed to Inform and guide the reader In the further clarification of the 
goal so that appropriate measurement procedures may more thoroughly 
Gpeclf led- 
Knowledge 

If a goal has been categorized as a knowledge goal» the designated goal 
expert (s) will need to be consulted for clarification along several lines. 
They should be asked for references to particular textbooks or other mater- 
ials that present such knowledge. This Information will help provide the 
type of content reference needed for developing the content part of the 
measurement specifications. Also the experts should be asked to note any 
key terms (e.g.> checks and balances) and/or major distinctions (e.g., 
among branches* not levels of government) that are salient. For more 
detailed examples, see the "General Clarification" sections In the Knowledge 
examples presented In the Appendix. 

Another aspect of clarification may have format Implications. If one 
can get clarlfla tlon of the cltlzen^related circumstances of the use of the 
content, then one may match the use to the appropriate questlon-am^wer format. 
If the emphasis In the goal Is on circumstances In which the citizen Is to 
recall the knowledge, as needed In making an argument for or against a 
particular public policy or action, then the most appropriate question- 
answer format would seem to be one which calls on the person to produce the 
answer (e.g., fill-In and £:3say-type questions). On the other hand. If the 
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envisioned citizen-related use is primarily to judge the veracity of some- 
one else's claims (e.g., evaluating a candidate's a«ssertions or a newspaper's 
story) > then answer-selection formats (e.g., true-false and multiple choice) 
would seem to be most appropriate. For more elaborate examples of the rela- 
tions between envisioned citizen-related use and format, study the "Relation 
to citizenship" and "' ^v^^c^^^io^s measure specifications'* sections in 
the Knowledge examples presented in the Appendix. 

In addition to format implications, clarifying the citizen-related 
use of the goal content provides a context which may be worked into the 
content of the measure. For example, a question concerning checks and 
balances might be set in the context of deciding whether to support an 
increase in executive authority. Additional examples of implications are 
presented in the Knowledge examples in the Appendix. 

Finally, the goal expert(s)' clarification may suggest that the 
type of information implied by the goal is more opinion than fact. That 
is» it may seem inappropriate to think of scoring the related answers as^ 
correct or incorrect. If this is the case» either one may tnerely tally 
answers and report frequencies for all answers, or one may overtly consider 
the goal to be about an attitude rather than knoffledge and use procedures 
discussed in this paper's section on attitudes. See Knowledge Example 1 in 
the Appendix for an illustration of this circumstance. 

Additional general guidance for developing measures of knowledge is 
widely available. Two such references are Ahman and Clock (1971, pp. 81- 
182) and Educational Testing Service (1959). For more detailed coverage, 
see Thomdike (1971, especially Chapters 3, 4, 5, 7, and 10). 




Skills 

If a goal Is considered to be a skill goal — I.e., Involving 
performance of a task when asked — clarification should be obtained from 
the goal expert(s) concerning the task(s) Involved. Particular topics 
of clarification which will be helpful In developing specifications for 
measures are descriptions of (1) each of the steps necessary to accomplish 
the task and their sequence, as appropriate (e.g., Skill Example 1 In the 
Appendix) , (2) any resources that should be available for use In the task 
(e.g.. Skill Example 2 In the Appendix), (3) special characteristics of 
the setting In which such a task nilght be carried out (e.g.. Skill Example 1 
In the Appendix), and/or (4) the outcome or product of the completed task 
(e.g.. Skill Example 2 In the Appendix). Mo^c generally, for some skills 
It will be appropriate to observe the process (see "Implications for 
Measure Specifications" In Skill Example 1 of the Appendix), vfalle with 
others the focus Is more appropriately placed on the outcome (see Skill 
Exampl ; '* in the Appendix). 

As with knowledge goals, the appropriateness of the measure Is 
Increased If the substance of the task and/or the setting for It can be 
related to the citizen use of the skill. Thus, the goal expert(s) should 
be asked to clarify when the skill would be used by a person In the citizen 
role. The reader should study the relationship between the **Relatlon to 
Citizenship** and Implications for Measure Specifications" sections of the 
Skill Examples In the Appendix, for e^tamples of measurement iu;i;>licatlons 
of the citizen use clarifications. 
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If, after clarifications from the goal expert(s), a primarily verbal 
measure Is Implied, then the goal Is probably better categorized and 
treated as a knowledge goal* As presented In the "Implications for Measure 
Specifications" section of Skill Example 2 In the Appendix, a question that 
asks for a list of things to do, measures knowing what, rather than how — 
l^e*, a knowledge, rather than a skill* as those terms are used In this 
paper* Further, asking for a discussion of why one did something would also 
be considered a knowledge measure, similar to Knowledge Example 1 In the 
Appendix. 

Specifications for skills measures should always Include procedures for 
informing the person being tested of what is expected, for motivating the 
performance, and for supplying the needed resources. See the "In^lications 
for Measure Specifications" sections of the Skill examples in the Appendix 
foiT illustrations* If these conditions do not hold, it Is likely that 
factors other than the skill will enter Into the behavior observed* If the 
goal expert(s) indicate that the goal refers to whether or how well persons 
usually do the task, rather than hoWyfell"tlTey' "argnrat^able of doing the 
task, then the goal should be categorized and treated as a behavior goal. 
Skill Example 1 in the Appendix provides a more concrete discussion of the 
differences between skill and behavior measures* 

Finally, It may facilitate data collection to use aggregate, rather 
than individual, measures, wh<^re possible* When the purpose In using 
the measure is to make decisions about a citizen education prc^icvji, 
rather than about individual persons, there is no need for obtaining an 
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Individual's score* Scores may be obtained for groups of persons, as 
Illustrated In Skills Example 1, In the Appendix. A more general dls^ 
cusslon of the opportunities for aggregate measures Is presented In a 
later section of this paper, entitled ^'Practical Considerations." 

Additional guidance for developing skill measures Is available. 
Flt2patrlck and Morrison (1971) provide one extensive coverage of the 
topic, and the CAFT Newsletter (Clearlngjioua for Applied Performance 
Testing) provides a regular updating. 
Attitudes 

If a goal Is categorized as an attitude goal — referring primarily 

3 

to what people say about their normal perceptions and/or evaluations **** 
one should encourage the goal expert(s) to consider whether the persons 
whose attitudes are to be measured are likely to be sufficiently aware 
of their attitudes to respond validly when asked directly about them. 
Host methods for measuring attitudes Involve asking directly, though 
there are Indirect methods, which are usually referred to as **projectlve 
measures? — Projectlves are used when there Is reason to believe that 
the attitudes are only revealed with Indirect tasks, such as word 
association and creative story telling. Projectlves are not used gener- 
ally In education, perhaps because the Inferences required are based on 
relatively elaborate and/or unsupported theory. An occasion for the 
Indirect approach Is presented In Attitude Example 1 In the Appendix. 

No distinction Is made here among the variety of labels used for 
such Internal states as "attitudes," %ellefs,'* '*drlves,'* 'Interests," 
•'opinions,** and'Values.*' 
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In addition to inaccessibility of the attitude, another reason that 
asking directly is often considered inappropriate is the suspicion that 
the person being quesitoned may not "tell the truth." There are several 
procedures for tainimizing and/or checking this suspicion. Firsts the 
introductory instructions may encourage honesty by having respondents 
omit giving any self-identification, or» where anonymity is not possible^ 
by assuring confidentiality of response. Second^ one may add questions 
that on one could be expected to answer in the desirable way» unless they 
were not *'telling the truth" (e-g.» answering "Always" to "I get all the 
information on the candidates before I vote"). A person who answers in 
the way d^asignated as not telling the truth on several such questions 
would not be included for scoring in the other items. Finally^ in such 
circumstances as are described in the following paragraph* giving an an- 
swer can involve a choice from among equally desirable — or equally un- 
desirable — alternatives » thus eliminating choices based on presenting 
oneself positively^ or negatively^ respectively, (see Scott, 1969» 
pp. 238*245 for further discussion of these procedures.) 

Among direct methods of measuring attitudes there are two basic 

types. If the strength of the attitude is likely to be manifest most 

clearly in direct comparison with other attitudes^ then the method used 

should reflect that, (see Attitude Example 2 in the Appendix for an 

illustration.) Most attitudes are measured^ however^ without comparison 

4 

to other attitudes. Xhus» it is important to determine with the goal 
expert(s) whether the attitude is seen as being strongly interrelated 
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with one or more other attitudes. The goal experts* views of the relation 
of the attitude to citizenship may help clarify this Issue. 

The cXarlflcatlou concerning citizen reXatedness of an attitude 
should also help In other ways to specify the appropriate phrasing of 
direct measures of the attitude. It may help to clarify whether the 
attitude Is In present^ or ongoing^ time» or whether It Is about the 
future. Compare Attitude Examples 2 and 3 with 4 In the Appendix for their 
different time references. It may also help to determine the attitude 
focus: sense of obligation^ belief^ feelings or behavioral orientation. ^ 
Compare Attitude Examples 2» 3» 4» and 5 for different attltudlnal 
focuses. Finally^ the clarification of citizen relatedness of the 
attitude can suggest appropriate content for the ineasure; study all atti- 
tude exa^les In the Appendix for Illustrations of content Implications.. 

Additional general guidance In developing the types of measures 
Indicated In this section may be found In Henerson* Morris & Fltz-Glbbon 
(1978» pp. 84-91) and Scott (1969» pp. 218-245). 



Direct measurement of attitudes does» however^ often Involve use of 
both positive and negative phraslngs of the attitude being measured. The 
purpose of this procedure Is to deal with the unthinking checker of choices » 
who will receive a midpoint or neutral score when half the Item phraslngs 
are positive and half are negative, (see Scott» 1969» pp. 239-241 for a 
further discussion.) 

^If actual behavior Is the concern^ the goal should be categorized and 
treated as described in the following section on "Behavior.** There are 
exceptions to this rule» as also discussed in the Behavior section and 
exemplified by Attitude Example 2 and 5. 
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Behaviors 

If a goal Is categorized as a behavior — referring primarily to 
what people do under natural clrcuiastances» as the term Is used here* — 
then the measurement procedure will Involve some type of observation. 
Direct observation of the behavior Is the most straightforward of the 
procedures. It Involves observation of particular aspects of behavior 
which have been clearly specified as occurring under explicitly defined 
circumstances. Thus» clarification should be obtained from the goal 
expert(s) concerning the specific aspects of behavior and their circum- 
stances of occurrence. Particular points of such clarification as 
discussed below are the bases for deciding the appropriate measure 
procedure and are Illustrated In the '^General clarification** and "Rela- 
tion to citizenship" sections of the behavior examples In the Appendix. 

Although direct observation Is the most trustworthy procedure, It ' 
Is appropriate only when the occasions of the behavior can be, antici- 
pated and localized and when the observer can be relatively unobtrusive. 
Compare Behavior Example 1 with other behavior examples In the Appendix 
for the conditions described here. 

When direct observation Is not appropriate, then one may be able to 
draw on the observations made by others who have some special knowledge 
of the particular aspects of behavior under the specified circumstances* 
These observations may be obtained directly from the others through 
structured Interviews or (questionnaires (e.g.. Behavior Example 2) or 
they may be found among official records (e.g., Behavior Example 3). 
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Whatever type of observation Is uS'^d* It Is of course important 
that precautions be taken to avoid biased observations. In addition to 
being aware of possible biases In the choice of observers > one should 
try to arrange for more than one observer; averages, or other summaries, 
of observations from the several observers are generally considered to 
be more reliable than any one observer's observations. 

Another Important Issue Is the need .to focus on each Individual 
among those whose behavior Is being measured* Because the purpose assumed 
In this paper In using the measure Is to make decisions about a citizen 
education program, rather than about Individuals Involved In the program, 
there ^l^^.usually no need to obtain Individual scores. Although an 
exception^ Is presented In Behavior Example 3t the other behavior examples 
In the Appendix are amenable to being carried out either with a repre- 
sentative sampling of the persons In the program or by using measurement 
procedures which do not allow for Individual Identifications. Behavior 
Examples 4 and 5 Illustrate this latter type of procedure. The measure 
In Behavior Example U depends upon the participants* observations of the 
settings In which they participate, commonly referred to as '^climate** 
scales. The measure In Behavior Example 5 Is commonly referred to as a 
"trace'* measure, which does not involve observing the behavior Itself, 
but only the evidence that the behavior has occurred. 

There are circumstances in which none of the above procedures is 
both sufficiently complete and practical. Under these circumstances, 
the person's own self-report of the behaviors might be the best possible 
way to obtain the appropriate measurements. Because the self-report 
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procedure has been discussed In this paper under the previous ^'Attitude" 
section, the reader should refer to that section and to Attitude Exam- 
ples 2 and 5 for additional Information. 

Additional general guidance for developing direct observational 
procedures may be found In Welck (1969, pp. 401-435). Guidelines for 
developing reports by others may be found In Anastasl (1968, pp* 419-421). 
General commentary concerning trace and other similar measures may be 
found In Webb, Campbell, Schwartz, and Sechrest (1966). 

Important Additional Concerns 

There are concerns beyond Individual goal clarification and related 
measure specifications that are Important In the overall effort of obtain- 
ing measures of citizen education goals. These concerns and related 
suggestions have been labeled as either practical or prudential, and are 
presented in the following sections. 
Practical Concerns 

Most standard references on testing, measurement, observational pro- 
cedures, and evaluation will indicate that a measure will not be depend- 
able if It consists of only one or a few Items or observations* These 
references generally contain rules of thumb for how many items are needed, 
contingent upon the type of measure and nature of the content. Of course, 
the vooTe that are needed for a dependable measure, the more time and 
effort are required to develop, put Into operation, and score the measure. 
If large amounts of time and effort are required, this can present a problem, 
for there are generally not just one, but several, goals to be measured. 
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If the amount of time and effort needed Is of concern, the person 
responblble for guiding the measure selection and/or development may be 
able to recommend a sampling procedure. The sampling Involves giving 
only some of the Items (or making only some of the observations) for 
any one Individual, but covering the total set of Items (or observations) 
across all Individuals. The procedure, referred to as **ltem sampling**^ 
or "matrix sampling,** was developed In the realm of knowledge measures, 
but may be more generally applied In the other categories of measures 
discussed In this paper. Husek and Slrotnlk (1967) provide basic guide- 
lines for use of this procedure. 

The use of the sampling procedure Is defensible only when the pur- 
pose of the measurements Is to make decisions about a citizen education 
program, not about Individuals. These decisions may be made on the basis 
of aggregate data and there Is no need to have Individual scores* Al- 
though there Is some discussion of aggregate measures In the sections on 
**Skllls** and **Behavlors»** the sampling {Procedures mentioned here can also 
be used for measures of knowledge and attitudes. 

A related consideration concerning the use of aggregate results 
Is that It Is not necessary for the measures to be as reliable as they 
would need to be for decisions about Individuals. This circumstance Is 
fortunate, because measures In the citizen education area In general 
may be expected to have reliabilities that are too low to serv« as bases 
for decisions about Individuals (Sanders, 1978). 
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Prudential Concerns 

There should be a concern for the well-being of the people being 
measured and those responsible for their education. Although in program 
assessment there is no need to report individual scores, there remains the 
possibility that an individual's answers may be identifiable as ones he/ 
she gave. Insofar as the particular measure may reveal information which 
may be used in a way that is harmful to the person, there is cause to con** 
sider procedures that avoid that possibility. Foremost among these proce- 
dures is allowing the questions to be asked or observations to be made with- 
out identifying the person. Other precautions also should be taken to 
prevent others from seeing the answers of, or recorded observations on, 
the individual. 

Reports of f?23r^ated information may also result in harm, insofar as 
groups — e.g., teachers in a given school — may be punished for the 
particular results of the measurement — e.g., poor performance by students' 
in the school. Thus, identification at the aggregate level should, if 
possible, be avoided. 

Finally, a major prudential concern is the community acceptance of the 
measurement effort related to the citizen education goals. Such measures 
may well include items or observations with which some members of the 
community will take ssue. The person responsible for the measurement effort 
should seek authorization of the content and procedures used, from a recog- 
nized legitimate office or body. 
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APPENDIX 

EXAMPLES OF GOAL CLARIFICATIONS AND RELATED MEASURE IMPUCATIONS 



Introduction 

This appendix contains examples of citizen education goals, cXarl* 
flcatlon of those goals, and related Implications for measure specif lea* 
tlons. The Information presented In the examples was selected and/or 
contrived to Illustrate Issues raised In the text of the **Gulde" section 
of this paper. 

However, ther€£ was also an attempt to develop realistic citizen educa* 
tlon goal-related Information. Th£ goals were selected from listings by 
others; specifically, sources Included lists of state goals (Delaware, 
1975; New Jersey, 1972, pp. 36-37; and Pennsylvania, 1978), and goals 
developed In the National Assessment of Educational Progress (Fair, 1975, 
p. 101), the International Evaluation of Educational Achievement (Toriey, , 
1978, pp. 3-A) and the Research for Better Schools Citizen Education pro- 
ject (Hill, 1978, pp* 80-82)* The selections of the particular goals 
from those sources were Intended to Indicate the broad range of content 
occurring normally In the area of citizen education* The goal clarifica- 
tion, presented In the "General clarification" and ''Relation to citizen- 
ship'^ sections of each exankpleT was developed by the author as plausibly 
elaborating on the goal, as well as serving the purpose of Illustrating 
Issues raised In the text of the ^^Gulde'^ section of this paper. 

In the anticipated use of the Guide, the sources of goal-related 
Information would, of course, be different from that described above. 



V 
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The goal would be one developed to Indicate a particular aspect cf 
citizenship valued by the goal developers* The goal clarification 
would result from an Interchange between the designated goal expert(s) 
and the person responsible for obtaining measures, based on the guld* 
ance offered In the ^'Ctj^Jldt. section of this paper* Then the latter 
person would be responsible for developing the ^'Implications for 
measure specifications,*' which would be taken to the goal expert(s) for 
further development or selection of the appropriate measure* 
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Knowledge Example 1 



Goal* : Students should recognize basic governmental purposes. 

General clarification* : The "basic governmental purposes" are the 

functions served by any form of governance: providing for social 
order and the well-being of those who comply with the social 
order / "Recognizing" these purposes means understanding that with- 
out government of some kind* there would be disorder and conflict. 
The goal refers not only to national^ state, and local government, 
but also to rules and authorities in large gathering places such 
aa schools and businesses. 

Relation to citizenship* : As citizens, people should respect the 
authority of government over more individual preferences and 
practicalitiea. This goal refers to a knowledge basis for 
that respect. That is, when isaues of government respect versus 
indi vidual conv enience arise, the purpose of government should be 
remembered* thereby providing support for respect for government. 

Implj>>cations for measure specifications : The question should depict 

situations in which some governmental regulation poses an inconven^ 
ience to an individual. The question would then call for a produc- 
tion-type answer giving a general rationale for obeying the govern- 
mental regulation. Choice of regulations should include those that 
are very local (e.g., a school regulation) and those more general 

*See the introduction to this appendix for a description of the 
sources of the goal-related information presented here. 
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(e*g*» a state traffic law)* Answers should Include some descrip- 
tion of social order» to be scored as correct* 

Another type of measure Is required to relate more directly to 
respect for government^ which Is an attitude rather than knowledge* 
The reader should see the section on attitudes for a discussion of 
appropriate measure specifications* 
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Knowledge Example 2 

Goal*: Students should be aware of International organizations and 
organizations functioning transnatlonally. 

General clarification* : The organizations are the ones that a person 
Is most likely to find In the news concerning political^ economic^ 
health, and social justice Issues. Being ''aware" refers to 
associating the names of the organizations to their purposes, In 
general terms. 

Relation to citizenship* : As a citizen, a person should be able to 
Identify a relevant organization Involved with a certain Issue, 
In order to facilitate finding additional Information about the 
Issue . 

Implications for measure specifications : The question should pose an 

International or transnational Issue, chosen from those most likely 
to be In the news. The question would then Include a request for 
the name of an organization that would be Involved across national 
boundaries In the Issue. In order to make the cltl^en^^related use 
of the knowledge explicit, the question should also Indicate that 
the organization's name could then he used to obtain further 
Information about the Issue. 



*See the Introduction to this appendix for a description of the 
sources of goal'^related Information presented here. 
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Knowledge Example 3 

GoaX^* : Students should know the history of the nation as necessary for 
responsible citizenship* 

General clarification* : The historical events that are related to this 
goal are found In the U.S. history texts used In the schools 
which have this goal. The focus should he on those events which 
clearly present the Issues Involved in developing the government 
of the U. S., e.g.» the development of the U. S. Constitution, 
selected Supreme Court cases, reaction to secessionist iDovements» 
and the government's relation to the economy. Such information as 
names and dates of Presidents, military conflicts* and technologi- 
cal innovations is not included per se. 

Relation to citizenship* : As citizens > people should Inform their 
representatives about and/or vote on the basis of their evalua- 
tions of particular governmental policies. They should be able 
to connect present policy proposals with events of the past, 
in order to use the lessons of the past to evaluate the present 
policy proposals. 

Implications for measure specifications : The questions should require 
association of governmental policies that are currently at issue 
in the news, with a past event In which the Issue was previously 
raised. The question should state a current issue and ask for 

^See the introduction to this appendix for a description of the 
source of the goal^rei&ted information presented here. 
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production/ recall of a similar past event that could provide 
helpful knowledge In resolving the current Issue. The partlc** 
ular current Issues used should be chosen for their clear re-* 
latedness to at least one of the salient events presented In 
the texts as discussed under ''General clarification" above. 

An Introduction to the question should Indicate the citizen 
use of the knowledge. That Is* It should point out that the 
Issue Is one about which people as citizens take a stand. 
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Knovledge Example 4 



Goal* : Students should recognize how different civic policies may affect 
people*s efforts to meet their economic needs* 

G^eral clarification* : **Clvlc policies** are general action orientations 
taken by any level of government or any of the levels operating 
In concert* Of particular Importance^ are policies having to do 
with employment assistance^ business stimulation, environmental 
regulations, federal reserve, civil rights, and product and service 
regulations. The **economlc needs** referred to are those of Individ** 
uals. Including especially the availability of various types of 
jobs, the purchase of products and services, the support of de* 
pendents, and the saving or Investment of money. 

Relation to citizenship* : As citizens, persons should be able to evalu- 
ate policy choices Intelligently, In order to vote and otherwise 
responsibly inform their representatives* Thus, they should 
realize the Implications of particular policies for how they and 
others In the society will meet their economic needs. 

Implications for measure specifications : Each question might provide a 
description of a recent or proposed policy from among those areas 
presented In the **General clarification** above. The question 
should then provide a set of possible answers (l^e., multiple choice) 
that represent desired outcomes from among the types of economic 

*See the Introduction to this appendix for a description of the 
goal-related Information presented here. 
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needs presented In "General clarification." The test taker would 
be asked to choose the most likely economic outcome of the 
policy given, A citizen-related context may be created If the 
question Indicates that the outcomes are ones claimed by different 
political candidates, and In order to determine which candidate 
to support, the citizen must evaluate the claims for their rela- 
tive correctness. 
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Skill Example 1 



Gpa^*: Students should be able to apply democratic procedures In 
small groups. 

General clarification* : The *'democratlc procedures'* are (1) allowing 
each person a reasonable opportunity to have her/his position 
on the Issues heard by the others, (2) Insuring that Individuals 
are not jeopardized for holding their positions, (3) making a 
dccislca on the basis of the majority, and (4) following through 
on the decision by everyone. **Small groups*' refers to committee^ 
like groups of from three to eight persons, brought together 
especially to work on some problem or to carry out some task. 

Relation to citizenship* : As citizens, persons may be Involved In 
political campaign committees* student governments, and social 
Issue groups. In which group decision making will probably occur. 

Implications for measure specifications : A small group setting, as 
defined In "General clarification** above, should be used. The 
groups should be given various tasks considered to be of 
the type mentioned In the **Relatlon to citizenship** section. 

Group members should be told the four particular democratic 
procedures wb^ch will be observed, and they should be encouraged 
to follow those procedures while working on the task. Otherwise, 
behavioral tendencies beyond possession of the skill Itself might 

^See Introduction to this appendix for a description of sources 
of the goal^related Information presented here. 
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become manifest. The group probably should be specially formed 
for the task, rather than being a naturally occurring group, 
e.g., of friends In a classroom. (Qf course, there should be a 
consistency In the basis for formation across all groups Involved.) 

The procedures as stated In the '^General clarlf Icatlon'^ sec- 
tion Imply that the measure Is an aggregate one. I.e., a measure 
of the group, not the Individual group members. Thus, though the . 
actions of Individual members are noted, the observations of each 
member need not be as extensive as It would be for an accurate 
Individual assessment. Another Implication of this circumstance 
Is that the task could be a limited one. Involving the group for 
only enough time to observe the four procedures, probably 10 to 
30 minutes, with the longer times required for a larger group. 



ft*. 
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Skill Example 2 

Goal* : Students should possess basic skills in obtaining information. 

General clarification* : "Basic Skills in obtaining information" includes 
being able to use the table of contents and index of a bookt to 
use 3 card catalog and shelving system in a library, and to question 
persons who know the subject of concern, in order to obtain informa- 
tion. These skills are probably processes of thinking of an appro- 
priate key word or phrase, using alphabetical and numerical order 
knowledge, noting what is found, and changing to another key word 
or phrase, if necessary to continue the search. The outcome of 
use of the skill is, of course, the possession of the information 
sought . 

Relation to citizenship* : As citizens, people are expected to obtain 

information about political candidates before voting and , 
about social policy issues in order to determine their positions 
on the issue. 

Implications for measure specifications : The presentation of the task 
to the persons whose skills ^re being measured should indicate 
clearly what information is to be sought and what resources are 
available for the search. (Otherwise, aspects other than the skill 
itself will be measured.) The information to be sought should be 
the type commonly related to the citizen role, e.g., a political 
candidate's views on assistance to the poor. 



See the introduction to this appendix for a description of sources 
of the goal^related information presented here. 
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Although a process of obtaining Information Is outlined in 
the ^^General cldrlflcatlon** section above, a close observation of 
It Is probably not possible* . Also, the test takers* verbal produc- 
tions of the process, or any other such listing, cannot logically 
be considered as a demonstration of the possession of the skill, 
because adequate verbalization may be neither necessary nor 
sufficient for carrying out the task* A viable alternative Is a 
focus on task completion; the time required to obtain the Informa- 
tion could be recorded* 
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Attitude ExampJ-g ^ 

Goal* : Students should appreciate the worth of all people. 

General clarification* : Thla goal refers to a positive, basically 
trustful feeling toward other people. It Includes all people, 
but the emphasis Is on persons different from oneself (e.g.^ of 
another race, religion, or nationality) and on those who are 
usually given lower status In the society (e.g.« children, elders, 
the poor, and the physically or mentally handicapped). 

Relation to citizenship* : As citizens In a democracy, people need to 

share the feeling underlying the belief that people who are differ- 
ent from themselves have viewpoints which deserve to be represented 
in the society. 

Implications for measure specifications : The emphasis on a feeling 
that underlies belief suggests a basic emotion which may be 
difficult to assess through response to direct questions or 
statements. A less direct orxentatlon that seems to be appropriate 
Is a structured word association technique called the Semantic 
Differential (Henerson, et al., 1978, pp. 89-91). Using this 
technique, the particular names of the groups mentioned In the 
^^General clarification** section above would be qulctdy associated 
by the respondent with ^ point along each of eight to ten scales. 
The scales should each be defined by bipolar adjectives, selected 
to represent various aspects of feelings related to worthiness, as 

See the Introduction to this appendix for a description of the 
sources of the goal-related information presented here. 
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.presented In the "General clarlf Icstlon" and "delation to cltlzea 
ship" sections above (e*g*, trustworthy - untrustworthy,) ♦ The 
scale points selected by the respondents could be combined to 
form scores for each groups each scale across groups* and across 
both scales and groups* 
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Attitude Example 2 

Goal* : Students should be loyal to country, to friends, and to others 
whose values they share. 

General clarification* : Loyalty means defending social Institutions 
and groups of people of which one Is a part. The defense may 
come In the form of speaking In support of, contributing. money 
or time to, and physically fighting for the group or Institution. 
It usually requires some sacrifice of personal Interest such as 
social status or physical security. 

^elation to cltli'enshlp* : As citizens, people are expected to be loyal 
to their national, state, and local governments as manifestations 
of democratic Ideals, and to other Institutions and groups, such 
as the school and the family, as they provide education and support 
for democratic government. 

Implications for measure specifications : According to the '^General 
clarification" above, loyalty Is a behavioral orientation which 
usually requires sacrifice of other, more self-interested orlenta* 
tions under conditions In which one*s own groups or Institutions 
are under attack. The ^'Relation to citizenship*^ section points 
out the related groups and Institutions^ Although the content of 
this goal Is clearly behavior* such behavior Is too Infrequent and 
too often hidden from others* views to be subject to sufficient 
observation. Instead, a self-report procedure Is used, and the goal 

*See the Introduction to this appendix for a description of the 
sources of the goal-related Information presented here. 
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becomes classified as an attitude, In concert with this paper^s use of that 
term* An appropriate procedure Is one based upon an Item form used by the 
Pennsylvania Department of Education In Its Educational Quality Assessment 
of citizenship: A situation, a behavior, and several mitigating circum- 
stances are presented, and the respondents Indicate whether they would engage 
In the behavior under each of the circumstances* For measurement of this 
attitude the situation would be one In ^Ich one of the groups mentioned above 
Is under attack, the behavior would be the manifestation of loyalty, and the 
mitigating circumstances would be various sacrifices of status or security 
resulting from the behavior. In an attempt to find respondents who are 
answering on the basis of social desirability, overwhelming mitigating circum- 
stances could be used on three or four of the Items, as the basis for a 
bullt-ln lie scale. 
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Attitude Example 3 



Coal* : students should believe that each person's civic behavior Is 
Important • 

General clarification* : **Clvlc behavior" means any activity relating 
primarily to rules and responsibilities In the Institutions of a 
democratic society, Including voting, taxpaylng, obedience to 
lavs, availability for public office, taking public position on 
Issues, and educating for civic behavior. Many of these behaviors 
should be seen In all of the Institutions of a dentocratlc society, 
Including schools. To **belleve" means to express to others the 
conviction, and *'ls Important" should be taken to Imply that the 
full functioning of the government Is dependent upon the civic 
behavior of every Individual. 

Relation to citizenship* : As citizens, people who hold this view are 
more likely to behave according to It and to encourage others to 
do so. Such participation Is necessary If the society Is to be 
democratic. 

ItttPllcatlons for measure specifications : Statements Indicating belief 
In the efficacy and other values of the civic behavior Indicated 
In ''General clarification'* above could form the basis of an appro- 
priate measure. Respondents would be asked to Indicate their 
agreement or disagreement with the statements. To prevent the un-^ 
thinking respondent from appearing to be totally In support of the 



See the Introduction to this appendix for a description of the sources 
of the goal-related Information presented here. 
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belief* some statements mav be phrased so that agreement would 
indicate a lack of belief; if half the statements are phrased 
each vay» then this unthinking respondent would be scored as 
having a midpoint* or neutral position on the topic* 
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Attitude Example A 



Goal* : Students should recognise the need for Individuals and cotomunl- 
ties to make wise use of resources* 

General clarification* : The ''resources** of concern are oil and other 
Sources of £uel» trees and other sources of building materials* 
and arable land as the source of food and much clothing. 
Exactly what policies constitute %lse use'* Is the topic of much 
debate and ls» thus» not specified here* However^ It Is Important 
that young people have a sense of obligation to question reflectively 
the wisdom of the various uses of these resources as they come to 
have responsibilities for managing their own» their family' s» and 
their community's use of them. 

Relation to citizenship* : As citizens* people are responsible to the 
public good» which iraplles actions for the basic well being of 
future generations* The sense of obligation for a wise use of 
resources underlies such actions. 

Implications for measure sj)eclflcatlons y According to the' "General 
clarification^ presented above» the goal refers to people's 
sense of obligation to question resource use when they take 
on adult responsibilities. A direct measurement approach seems to 
be the only approach that is appropriate for this cognitlvely^ased» 
future-oriented attitude. The respondent's agreement or disagree- 
ment would be sought to a set of statements which expressed the 



it 

See the introduction to this appendix for a description of the 
sources of the goal-related information presented here. 
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obligation^ and lack of obligation, to a questioning orientation 
concerning use of resources and settings mentioned above* 
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Attitude Example 5 

Goal^ : Students should actively work for civic Improvement. 

General clarification^ : **Clvlc Improvement" Is anything which betters the 
conditions of public property or pubJLlc events or which Is Involved 
In dealing positively with social Issues* *Vork** here means going 
beyond any paid job that a person has, even If that job Itself Involves 
civic Improvement; also excluded are activities that the person Is 
forced to do by his or her authorities* 

Relation to citizenship^ : As citizens, people can be expected to volunteer 
a part of their time occasionally to Improve the society. 

Implications for measure specifications : While this goal refers primarily 
to what people do, and not to what they say. It would be difficult to 
adequately observe related behavior, which may be assumed to occur In 
reSatlon to family, club, and/or church* as well as school. Thi)s» 
the measurement procedure most economical here is self-report, which 
is why the goal is considered an attitude goal, as the term attitude 
Is used In this paper* The self-report form would appropriately In- 
volve questions concerning Involvement In the type of activities suggested 
In the **General*cl&rlflcatlon** and **Relatlon to citizenship** section 
above* Besponses might be open-ended or structured. If appropriate 
levels of Involvement could be prespeclf led. _. 

See the Introduction to this appendix for a description of the sources 
of the goal-related Information presented here* 
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Behavior Example 1 

Goal* : Students should guard the safety and health of others. 

General clarification* : Students "guard the safety and health of 

others" by obeying the safety rules made for the school. These 
are (1) no running In the halls, (2) no riding of bicycled on the 
school grounds, and (3) wearing proper attire for physical educa- 
tion classes. 

Relation to citizenship* : As citizens, people should act In ways that 
do not endanger others. School safety rules are meant to convey 
this general behavioral tendency, as well as to provide for the 
safety of the students In the school. 

IiDpllcatlons for measure specifications : Direct observation of obedience 
to the rules mentioned above Is possible, because the behavior Is 
localized, the occasion for Its occurrence can be anticipated, and , 
the observation may be done unobtrusively. The "Relation to 
citizenship" section Implies that other safety-related behavior 
also should be observed; pushing In lunch lines could be con- 
sidered such a behavior. This behavior Is also amenable to direct 
observation. 



See the Introduction to this appendix for a description of the 
source of the goal-related Information presented here. 
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Behavior Example 2 

Goal* : Students should be ethical and dependable In work, school, and 
social situations. 

General clarification* : **Ethlcal** Is here defined as being honest. 
"Dependable*' means being on time and doing the activities agreed 
upon. 'Vork" refers to jobs, even temporary ones, for which the 
worker Is paid. "School" refers to both classroom and outside 
classroom settings. **Soclal situations** refers to Interactions 
with people not Included In the two other situations. 

Relation to cltlgeshlp* : As citizens, people should meet their 

obligations to others with whom they work or have other ongoing 
Interdependent relationships. In addition to family obligations. 

Implications for measure specifications : Direct observation of the 
behavlora presented here does not seem possible; they are not 
sufficiently localized, nor are they open to observation by an 
outsider. Instead, a questionnaire on the above topics could be 
given to teachers, employers of students, and students themselves, 
all of whom observe the natural occurrence of these behaviors. 
Although It Is possible to use this procedure to obtain scores for 
Individuals, It would be prudent to avoid Identifying Individuals 
and Instead to produce aggregate summaries across Individuals. It 
would dlso be practical to obtain such reports from as small a 
sample of respondents as Is technically feasible. 



*See the Introduction to this appendix for a description of the 
sources of goal-related Information presented here. 
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Behavior Example 3 



Goal* ! Students should comply with public laws. 

General clarification* : "Public laws'* are those that apply to all 

members of the public; **publlc law*' Is not Interpreted as Includ- 
ing school regulations that have no counterparts In laws which 
apply to all members of the public. Of particular concern lii 
disobeying public laws are assaults, property thefts, property 
defacing and destruction, possession of Illegal substances, and 
embezzlement . 

Relation to citizenship* : As citizens, people should obey the laws. 

Implications for measure specifications : The direct observation of 

Illegal behaviors is not an appropriate procedure because such be- 
havior Is not localized, not readily anticipated, and often does 
not occur l£ there Is any observer present. The relevant Information 
could be obtained through use of official records of convictions or 
admissions of guilt for commission of these crimes by the students 
In the school. Although a simple tally of such crimes might be 
sufficient for setting priorities, it may be helpful if any related 
program is planned, to have also a record ^£ particular individuals 
involved. However, see the "prudential Concerns'* section of the 
preceding paper for precautions to be taken In having such informa- 
tion in identifiable form. 

See the Introduction to this appendix for a description of the 
sources of goal-related information presented here. 
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Behavior Example A 

Goal* : Students shoulc shew respect and caring for others. 

General clarification^^ : "Respect" refers to treating persons with 

esteem* thereby making them feel that they are Important as Indlvi*^ 
duala* "Caring", means ^olng considerate things for people In times 
of need; caring does not here refer to romantic affection or friend* 
ship. ^'Others^^ Is used here to mean people In general* with 
special emphasis on people who are In the schools other students* 
teachers* clerical staff* lunchroom staff* custodians* admlnlstra* 
tors* and visitors. 

Relation to citizenship^^ : As citizens* people should support the 

physical and emotional well-being of one another. Giving respect 
and caring are major factors leading to physical and ^notional 
well^ielng. 

Implications for measure specifications : The behaviors Indicated here 
may occur at any time and any place* and* thus* are not good candl* 
dates for direct observation* In fact* as Indicated in the deflnl^ 
tlon of "respect" given above* It Is the Intact of the behavior* 
l*e** the perceptions of the others* that should be the focus of the 
measurement. That Is* within the suggested boundaries of the others 
mentioned above* those "others" should be asked whether they receive 
respect and caring from the students. As part of the development of 
Items for the measure* specific algns of respect and of caring should 
be sought from meuibers of these groups of others in the school. 



'^See the Introduction to this appendix for a description of the 
sources of the goal^related information presented here. 
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Beha ior Example 5 



Goal*: Students should recognize the value of protecting the envlronnt'nt. 

General clarification* : Underlying this goal Is a concern with Utter, 
wastefulness » and damage done to property and the biological envl- 
romoent by school children and young adults. 

Relation to citizenship* : As citizens, people should behave In ways 
which conserve the physical, biological, and aesthetic resources 
available for public use. 

ImffUcatlons for measure specifications : The focus of measurement here 
Is not to i:(apture the behavior Itself, but Instead to observe the 
aftermath or traces of the behavior: the damage done, the usables 
that are discarded, and the trash. In the school, these traces 
may be linked. If reasonable precautions are tak<±n, to the behavior 
of the students In the school. However^ tvaceu may be very diffi- 
cult to attribute to students when the traces are outside the 
school. In order to measure related behavior of students outside 
the school, the most appropriate orientation probably would he to 
depend upon what students say they do. The concerns for obtaining 
people's reports about themselves is discussed In the "Attitudes" 
section of this paper; an example Is given as A^^ltode Example 5 
In this appendix. 

it 

See the Introduction i.,- J appendix for a description of the 
aources of the goal-relate ^'^rmatlon presented here. 
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